With the proliferation of computerized automation of processes in every aspect of life, data storage and processing have become a major component of networked systems handling financial and other transactions. In such systems, data may be entered, modified, or deleted from a number of sources. The same data may be maintained in multiple data stores in same or different formats, and a data store may have to pick up or synchronize changes to data based on changes in a different store. Various data stores from simple tables to complicated databases may be maintained and synchronized as new entries or modifications are made by different sources. The changes may be synchronized at regular intervals. In many cases, the databases are partially or completely related.
When synchronizing data between databases, a typical solution is to maintain a detailed operation log at a source database and play back the operations in the original order during synchronization to a target database. The operation log keeps track of column level changes. That is, with every INSERT, UPDATE and DELETE operation, the original and new values for each column are maintained. These operations are then replicated or “played back” at the target database using the same sequence as determined by the operation log.
Over time the operation log may become very large. The operation log may be purged after the updates are played back on the target database, but this depends on being able to play back updates at a regular frequency. Also, a separate operation log table may have to be maintained for each table in the source database making the schema more cumbersome.